Quantitative softness and texture bimodal haptic sensors for robotic clinical feature identification and intelligent picking

Replicating human somatosensory networks in robots is crucial for dexterous manipulation, ensuring the appropriate grasping force for objects of varying softness and textures. Despite advances in artificial haptic sensing for object recognition, accurately quantifying haptic perceptions to discern softness and texture remains challenging. Here, we report a methodology that uses a bimodal haptic sensor to capture multidimensional static and dynamic stimuli, allowing for the simultaneous quantification of softness and texture features. This method demonstrates synergistic measurements of elastic and frictional coefficients, thereby providing a universal strategy for acquiring the adaptive gripping force necessary for scarless, antislippage interaction with delicate objects. Equipped with this sensor, a robotic manipulator identifies porcine mucosal features with 98.44% accuracy and stably grasps visually indistinguishable mature white strawberries, enabling reliable tissue palpation and intelligent picking. The design concept and comprehensive guidelines presented would provide insights into haptic sensor development, promising benefits for robotics.

The PDF file includes: Supplementary Text Figs.S1 to S22 Tables S1 to S4 Legends for movies S1 to S4 Other Supplementary Material for this manuscript includes the following:

Supplementary Text
Text S1.Theoretical analysis and sensing mechanism of three-dimensional forces perception To elucidate the mechanism and theoretical basis of three-dimensional force perception, a decoupling analysis of three-dimensional force signals is conducted using a piezoelectric module as a representative example.It should be pointed out that assuming a piezoelectric sensor has a linear relationship with the applied force, the conversion coefficient is 4.When the piezoelectric sensor is exposed to a normal force in the z-direction, its four dispersed sensing units react equally, resulting in similar changes in voltage across V1 to V4.This response is characterized by the transfer ratio 1 and conversion coefficient 4, where Vi = 14Fz for i=1-4.For shear forces in the xdirection, V1 (V3) and V2 (V4) exhibit equal but opposite voltage responses, as determined by the coefficient 2.Similarly, the voltage response ratio 3 is associated with shear forces in the ydirection.Therefore, the relationship between the output voltage and the applied force can be expressed as: The principle of superposition enables the determination of the voltage response for each piezoelectric module under three-dimensional forces, as detailed in equation (S1).Using this approach, the responses in channels V1 to V4 effectively decouple the three-dimensional force experienced by the sensor, as demonstrated in equations (S2) and (S3): Meanwhile, the output voltages along the x, y, and z directions will change with the applied normal and shear forces, which can be calculated by taking the average outputs of the four units as follows: The decoupled three-dimensional force remains closely related to the voltage output in both the normal and shear directions, as well as to the sensor's sensitivity, which can be expressed as follows: According to equations (S3-S5), the sensitivity along the x, y, and z directions can be obtained to further reveal the sensing mechanism of three-dimensional forces perception: The relationship between the normal and force components Fx -Fz and converted component Vx -Vz, which is also called the calibration coefficients kxkz, can be determined by experimental measurements.

Text S2. Theoretical analysis of softness measurement
The measuring mechanism of the elastic coefficient is based on a simple model to consider the elastic deformations of the sensors and the measured object.When an external force F is applied, the overall deformation x of the system is the summation of the deformations of both the sensor x1 integrated in the fingertip and the measured object x2, i.e., 12 x x x =+.According to force-displacement curve, the elastic coefficient of the sensor k1 and measured samples k2 under the force F can be derived as . Then the equivalent elastic coefficient of the whole system can be further expressed as . Therefore, the elastic coefficient of the measured samples satisfies 2 Text S3.Theoretical analysis of texture measurement A theoretical analysis based on haptic perception is carried out to reveal the regular texture measurement mechanism.In this analysis, a robotic hand applies a 1 N force to an object with uniformly spaced textures and slides at 4.5 mm/s, generating frictional vibrations.When a piezoelectric sensor encounters successive textures, it produces a regular pulsed electrical signal characterized by its main frequency f.The variation in the time intervals between the peaks of these signals, represented as Δt, can be utilized to calibrate the texture spacing ΔP.Thus, a relationship between f, ΔP, and the sliding speed v of the robotic hand can be obtained as: As the sensor moves smoothly across a sample at a constant velocity, with neighboring tactile units positioned at a specific distance d.The piezoelectric signals produced upon encountering the initial and subsequent gate textures are recorded at times t1 and t2, allowing for recording the time interval Δt.Consequently, the velocity at which the sensor traverses the object's surface can be described as: According to equations (S7-S8), a direct relationship between the texture spacing ΔP and the main frequency f can be established:

Fig. S1 .
Fig. S1.The sensing performance of the piezoelectric layer.(A) Voltage responses of the piezoelectric layer at vibration frequencies of 1000, 1100, and 1200 Hz in the time domain.(B) Response time (i.e., 0.6 ms) of the piezoelectric layer.(C) Static pressure sensing performance of the piezoelectric layer for an applied force of 1 N.

Fig. S2 .
Fig. S2.The sensing performance of the piezoresistive layer.(A) Voltage changes of the piezoresistive layer at vibration frequencies of 10, 20, and 30 Hz in the time domain.(B) Responserelaxation time (i.e., 79 and 132 ms) of the piezoresistive layer.(C) Static pressure sensing performance of the piezoresistive layer for an applied force of 1 N with 0.5 mm/min loading rate.

Fig. S3 .
Fig. S3.Working principles and corresponding FEA predictions of multidimensional sensing implementation.The piezoelectric potential distribution of the sensing array under (A) normal and (B) shear forces of 2 N.

Fig. S4 .
Fig. S4.The stress distribution of piezoresistive modules under the shear force.(A) The structural design of the piezoresistive module.(B) The stress distribution of piezoresistive sensing elements.

Fig. S6 .
Fig. S6.The discrimination accuracy of the bimodal sensor.The comparison between the standard force value and the force measured by the sensor under (A) normal and (B) shear forces.

Fig. S7 .
Fig. S7.The detection limit/range and pressure resolution of the bimodal sensor.The performance outputs of (A, B) piezoelectric and (C, D) piezoresistive modules were measured at the applied normal and shear forces.

Fig. S8 .
Fig. S8.The sensing stability of the bimodal sensor.Cycling tests of the (A, B) piezoelectric and (C, D) piezoresistive modules tested over 1000 cycles under normal and shear forces of 1 N.

Fig. S9 .
Fig. S9.The long-term stability of the bimodal sensor.The voltage output of the sensing modules based on (A, B) piezoelectric and (C, D) piezoresistive mechanisms upon normal and shear forces of 1 N, respectively.

Fig. S10 .
Fig. S10.The decoupling performance of the bimodal haptic sensor.The real-time output of the (A) piezoelectric and (C) piezoresistive modules.The decoupling accuracies of the (B) piezoelectric and (D) piezoresistive modules.

Fig. S11 .
Fig. S11.The sensing performance of the bimodal sensor in response to variations in temperature changes.The performance outputs of (A, B) piezoelectric and (C, D) piezoresistive modules were measured at the applied force of 3, 4, 5 N and 1, 3, 5 N, respectively.

Fig. S12 .
Fig. S12.The sensing performance of the bimodal sensor in response to variations in mechanical deformations.The performance outputs of (A, B) piezoelectric and (C, D) piezoresistive modules were measured at the applied force of 3, 4, 5 N and 1, 3, 5 N, respectively.

Fig. S13 .
Fig. S13.The decoupling performance of the bimodal haptic sensor in response to variations in mechanical deformations.The real-time output of the (A) piezoelectric and (C) piezoresistive modules.The decoupling accuracies of the (B) piezoelectric and (D) piezoresistive modules.

Fig. S14 .
Fig. S14.The deformation of the bimodal sensor.The strain distribution of the piezoelectric sensor in (A) normal and (B) oblique contact with grapes.

Fig. S20 .
Fig. S20.The sensitivity of the bimodal sensor.The voltage output of sensing modules based on (A, B) piezoelectric and (C, D) piezoresistive mechanisms varies when the applied normal and shear forces range from 0.1 to 7 N. Insets show the measured voltages under the maximum detection limit.

Fig. S21 .
Fig. S21.The sensing stability of the bimodal sensor with small dimensions.Cycling tests of the (A, B) piezoelectric and (C, D) piezoresistive modules tested over 1000 cycles under the normal and shear forces of 0.5 N.

Fig. S22 .
Fig. S22.Force versus displacement curve of PDMS bump on the sensor during peeling test at 0.5 mm/s constant peeling speed.